
<HTML> 
<HEAD> 
<meta content="text/html;charset=utf-8" http-equiv="Content-Type">
<link rel="stylesheet" href="nlp.css" type="text/css" media="all" /> 
<br>
<TITLE>NLP12 Project: Wikification</TITLE> 
</HEAD> 

<BODY> 
<h1>Project: Wikification</h1>
<b>Shimi Malka 066461641</b>
<br>
<b>Netali Alima 300712742</b>

<h2>Our Solutions: </h2>

This Project covers The process of building a model of wikipedia links and 
getting an article from the wikipedia clean the text from all wikipedia annotations
and recover the article according the model we learned.
<br>Finally calculate the Accuracy of this Decision process.
<br>

<h2>First Phase - Building the Train Data Set: </h2>
choosing X articles with more than N words and M links from the wikipedia:<br>
We started from a defined category and took all the links inside it and search for 
appropriate articles (that answer our conditions), It is a BFS scan.
<br>

<h2>Second Phase - Building the model & Training</h2>
The model is a cfd - (term,Link,no. of times the link appear for this term)<br>
For each article, go over all its links and fill the cfd accordingly.<br>
Notice that we are cleaning the term (the cleaning process will be explain later).
<br>

<h2>Third Phase - Building the Test data set:</h2>
Like building the train data set with the limitation of articles that appear in 
the train will not appear in the test.
<br>

<h2>Forth Phase - Decision process</h2>
After we build the cfd (for each term we have the list of links 
associated to this term in all articles in the training set, 
and for each link and term we have the no. of times the link was associated to this term).<br>
We implemented the BaseLine decision process that means to take the most frequent link 
which was associated to the term wanted.
<br>

<h2>Fifth Phase - Accuracy calculations</h2>
After building the test data, for each article in the test data set we build a real map each
link we find in the page (map of term-link) then we clean the text of the article, 
and performing the decision process above for all terms in the page, the decision 
process gives us a map of term-link of our decisions.<br>
Then we compare the 2 maps and count the no. of hits in and sums it up for all articles.<br>
The accuracy is calculated and returned - Accuracy = no. of hits/no. of links in all articles.<br>
<li>The cleaning process:<br>
First we are taking only the real text without all wikepdia annotations (using JWPL library).<br>
Then we are performing morphological analysis in Hebrew, that means to run the 
hebtokenizer python program, and then run Meni's thesis on the result that tagged 
each word in the article and helped us to split and stem phrases in hebrew.<br>
After tagging we have for each word a Lemma property that help us doing the split.<br>
After all that we removing all the words in the text that equal to one of the words in the he-stopwords.txt file.<br>
<br>

<h2> Conclusion:</h2>
We noticed that if the cleaning process not include the 
segmentation and stemming process and only include the remove stop words process, 
the accuracy of the decision is similar to the accuracy of the above process.<br>
The accuracy without segmentation&stemming&remove stop words process is 0.59 for 5000 articles in the train.<br>
The accuracy of the above process is 0.63 for 5000 articles in the train.<br>
At first we anticipated that we will have differences - with the segmentation & stemming 
the accuracy will be much higher - but the calculations shows us that the accuracy is similar.<br> 
<br>

<h2>Third party Packages</h2>
<li>JWPL - Wikipedia API for parsing and working with wikipedia dump.<br>
<li>Jython - For running python scripts in java.<br>
<li>Meni's PhD thesis program - For segmentation and stemming.<br> 

<h2>Running Examples:</h2>
<br>
Running on 5000 articles in the train and 1000 in the test (With segmentation&stemming&remove stop words):<br>
<pre>
Bulding wikipedia Train DataSet with 5000 articles.
  get so far : 10
  get so far : 20
  .
  .
  .
  get so far : 5000
[]
TAGGED
Reading corpus data from file...
Finish loading compact data from file
Start reading first similar words map from file
Finish successfully first similar words map initializtion from file
Start reading compact trigram prob model from file...
Finish reading compact trigram prob model from file
Loading PN model
pnmodel = tagger/splt
window size:5
max type:18
min location:-2
Done Loading PN model
Start reading BGU lexicon...
Finish reading BGU lexicon
--------Traning Done!----------
--------Start generating Test DataSet---------
--------Done generating Test DataSet---------
--------Start Decision process & Accuracy calculation---------
Accuracy = 0.6318332651459721
</pre>
<br>
<h2>Classes Explenations:</h2>
<li>WikiData - has the list of articles we gather from the wikipedia dump.<br>
<li>WikiCfd - has the cfd (term,link,no. of times the link is associated to this term).<br>
The function training is building the cfd according to the X articles we have in the WikiData.<br>
<li>WikiDecision - abstract class that responsible on the decision process of matching a link to a term.<br>
One abstract function decide that recieves a term and should return a string of the matching link.<br>
findTerms function that recieves clean text and return a list of terms that exists in the text.<br>
buildDecisionMap function that recieve a list of terms and match for each term the right link with the help of decide function and return a map of term-link.<br>
<li>WikiDecisionBaseLine - inherit the WikiDecision class and implement the function decide according to the base line.<br>
The base line is to take the link that have the maximal no. in the cfd for the term - means to take the most frequent link of the term we recieve.<br>
<li>Linguistic - static class that responsible of cleaning the text.<br>
Holds the tagger, hebTokenizer, BitResolver.<br>
The only public function is cleanText that gets the text without wiki notations and perform the Segmentation & stemming and 
remove stop words process with the help of the hebtokenizer, tagger and bitResolver.<br>
<li>Tagger - Meni's thesis, we copied the basic tagger class and change there a few things so we will be able to get the list of tagged sentences after tagging 
because we need to use the lemma property to do the segmentation.<br>
we added a function to get the tagged sentences after tagging.<br>
<li>HebTokenizer - class for runninf the python code hebtokenizer.py<br>
<li>BitMask - responsible of performing the segmentation itself.<br>
The function bitResolve gets the tagged sentences list and performing the segmentation with the help of the lemma property.
<li>Analyze - Responsible of testing the Decision process. It builds the test data set and calculate the Accuracy of the decision process.
<li>App - This is the main function.
</BODY>



